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t = o <- si 



initialize: p = 0, -? = 0,p = 0 4- S2 



initialize parameters: for i = 1...M; j = 1, K 
do g = 0, u\ = 1, b\ = 1, <4 = 0, 4 = 0, and </ = 1. 



observe s 0 , the system state at time 0 



S3 



S4 



S5 



The actor chooses an action a 0 based on its current parameters 9q and state s 0 . 
The probability of choosing each possible action is given by: P(a 0 = a) = 7r0n(so, a). 



t = t + l 



Implement the action a £ 



S6 



S20 



From Figure 2. 



Move to the next state s t+1 and observe the reward r t . S7 



Choose a new action q t+1 based on fl t and gt+ij ^(gt+i = a ) = ^(st+i^fl) ^~ S8 



Let /(at, s t ) = P(a t = a|s t = 5), /(a m , = P(a t+1 = a|s m = 5). 
Evaluate partial derivatives of /(at, s*) and f(a t+u s t +i) with respect to 
all parameters comprising the vector 9 t , 



S9 



Go to Figure 2. 

Figure 1: Part I of the ACFRL algorithm 
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From Figure 1. 



Arrange the partial derivatives of j(a u s t ) and f(a t +u St+i) 
into vectors X* and X t+1 , each of length n. 



S10 



Compute Q^(s u a t ) = ET=iP f *» ™ d Q*(sfrn,flt + i) - ISg^ 1 <- SU 



Update p: p t+i = p t + a t {r t - Pt)< 



H <- S12 



Update p: ft+i = + Qfrfo - ft + Qg* a m ) - Q*\{s u a*))*- \ <- S13 




Yes ^ACFRL(l)?^ Ko 



so or a t 



Yes 



No 



z t+ i = Xzt + V In 7r et (s t+1 , a f+ i) «- S16 



z t +i = zt + Vln7r et (s t+ i,at + i) <- S15 



Zt+i = Vln7rg t (3 t +i,fflt+i) <~ S14 



S18 | Update fl: g t+1 = g t - /? t Q^(s t+ i,a f+ i)Vln7re f (s t+1 ,a t+ i) | «- S17 
< ^For i = 1, n: (absolute value of ith component of Ot+i) > M z fy — 5?-, gig 



No 



Go to Figure 1 



ue of ith 



Set absolute val 
component equal to M± 



Figure 2: Part II of the ACFRL algorithm 
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Figure 3: Interference fuzzy labels used by the transmitters. 




Backlog 

Figure 4: Backlog fuzzy labels used by the transmitters. 



